Semi-Supervised Knowledge Amalgamation for Sequence Classification

Sequence classification is essential for domains from medical diagnosis to online advertising. In these settings, data are typically proprietary, and annotations expensive acquire. Often times, so few available that training a robust model scratch impractical. Recently, knowledge amalgamation (KA) has emerged as promising strategy models without this hard-to-come-by labeled dataset. To achieve this, KA methods combine the of multiple pre-trained teacher (trained on different tasks proprietary datasets) into one student becomes an expert union all teachers’ classes. However, we demonstrate state-of-the-art solutions fail in presence overconfident teachers, which make confident but incorrect predictions instances classes upon they were not trained. Additionally, to-date no work explored sequence models. Therefore, propose then solve open problem semi-supervised (SKA). Our SKA approach first learns estimate how trustworthy each given instance, rescales predicted probabilities teachers supervise model. solution overcomes through careful use very small amount instances. We beats eight alternatives four real-world datasets by average 15% accuracy with little 2% being annotated.